SCANN: Synthesis of Compact and Accurate Neural Networks

Deep neural networks (DNNs) have become the driving force behind recent artificial intelligence (AI) research. An important problem with implementing a neural network is the design of its architecture. Typically, such an architecture is obtained manually by exploring its hyperparameter space and kept fixed during training. This approach is time-consuming and inefficient. Another issue is that modern neural networks often contain millions of parameters, whereas many applications and devices require small inference models. However, efforts to migrate DNNs to such devices typically entail a significant loss of classification accuracy. To address these challenges, we propose a two-step neural network synthesis methodology, called DR+SCANN, that combines two complementary approaches to design compact and accurate DNNs. At the core of our framework is the SCANN methodology that uses three basic architecture-changing operations, namely connection growth, neuron growth, and connection pruning, to synthesize feed-forward architectures with arbitrary structure. SCANN encapsulates three synthesis methodologies that apply a repeated grow-and-prune paradigm to three architectural starting points. DR+SCANN combines the SCANN methodology with dataset dimensionality reduction to alleviate the curse of dimensionality. We demonstrate the efficacy of SCANN and DR+SCANN on various image and non-image datasets. We evaluate SCANN on MNIST and ImageNet benchmarks. In addition, we also evaluate the efficacy of using dimensionality reduction alongside SCANN (DR+SCANN) on nine small to medium-size datasets. We also show that our synthesis methodology yields neural networks that are much better at navigating the accuracy vs. energy efficiency space. This would enable neural network-based inference even on Internet-of-Things sensors.Comment: 13 pages, 8 figure

Efficient Neural Network Synthesis and Its Application in Smart Healthcare

Deep neural networks (DNNs) have become the driving force behind recent artificial intelligence (AI) research. With the help of a vast amount of training data, neural networks can perform better than traditional machine learning algorithms in many applications. However, there are still several issues in the training and deployment of DNNs that limit the use of DNNs in various applications. First, an important problem with implementing a DNN is the design of its architecture. Typically, such an architecture is obtained manually by exploring its hyperparameter space and kept fixed during training. Due to the need to navigate a search space based on a large number of hyperparameters, this can become challenging. This forces the space of possible architectures to grow exponentially. As a result, using a trial-and-error design approach is very time-consuming and leads to sub-optimal architectures. In addition, approaches like neural architecture search (NAS) based on reinforcement learning (RL) and differentiable gradient-based architecture search often incur huge computational costs or significant memory requirements. Another issue is that modern neural networks often contain millions of parameters, whereas many applications require small inference models due to imposed resource constraints, such as energy constraints on battery-operated devices. However, efforts to migrate DNNs to such devices typically entail a significant loss of classification accuracy. In addition, unlike the human brain that has the ability to carry out new tasks with limited experience, DNNs generally need large amounts of data for training. However, this requirement is not met in many settings, and hence is a limiting factor in the applicability of DNNs in several applications such as smart healthcare. To address these problems we propose neural network synthesis frameworks to generate very compact and accurate DNN architectures, optimize various hyperparameters of the DNN models, and reduce the need for large training datasets. We first introduce a two-step neural network synthesis methodology, called DR+SCANN, that combines two complementary approaches to design compact and accurate DNNs. At the core of our framework is the SCANN methodology that uses three basic architecture-changing operations, namely connection growth, neuron growth, and connection pruning, to synthesize feed-forward architectures with arbitrary structure. Then, we develop the CURIOUS DNN synthesis methodology. It uses a performance predictor to efficiently navigate the architectural search space with an evolutionary search process. To address the need for large amounts of data in training DNN models, we propose the TUTOR DNN synthesis framework. TUTOR relies on generation, verification, and labeling of synthetic data to address this challenge. We then apply the proposed DNN synthesis frameworks to smart healthcare. The emergence of wearable medical sensors (WMSs) alongside efficient DNN models points to a promising solution for the problem of disease diagnosis on edge devices, such as smartphones and smartwatches. In this area, we first propose a framework for repeated large-scale testing of SARS-CoV-2/COVID-19, called CovidDeep. CovidDeep combines efficient DNNs with commercially available WMSs for pervasive testing of the virus and the resultant disease. It achieves the highest test accuracy of 98.1% for a three-way classification among three, cohorts including healthy, asymptomatic (to detect the virus), and symptomatic (to detect the disease) patients. We also propose a framework for mental health disorder diagnosis, called MHDeep. MHDeep utilizes efficient DNN models and data obtained from sensors integrated in a smartwatch and smartphone to diagnose three important mental health disorders: schizoaffective, major depressive, and bipolar

TUTOR: Training Neural Networks Using Decision Rules as Model Priors

The human brain has the ability to carry out new tasks with limited experience. It utilizes prior learning experiences to adapt the solution strategy to new domains. On the other hand, deep neural networks (DNNs) generally need large amounts of data and computational resources for training. However, this requirement is not met in many settings. To address these challenges, we propose the TUTOR DNN synthesis framework. TUTOR targets tabular datasets. It synthesizes accurate DNN models with limited available data and reduced memory/computational requirements. It consists of three sequential steps. The first step involves generation, verification, and labeling of synthetic data. The synthetic data generation module targets both the categorical and continuous features. TUTOR generates the synthetic data from the same probability distribution as the real data. It then verifies the integrity of the generated synthetic data using a semantic integrity classifier module. It labels the synthetic data based on a set of rules extracted from the real dataset. Next, TUTOR uses two training schemes that combine synthetic and training data to learn the parameters of the DNN model. These two schemes focus on two different ways in which synthetic data can be used to derive a prior on the model parameters and, hence, provide a better DNN initialization for training with real data. In the third step, TUTOR employs a grow-and-prune synthesis paradigm to learn both the weights and the architecture of the DNN to reduce model size while ensuring its accuracy. We evaluate the performance of TUTOR on nine datasets of various sizes. We show that in comparison to fully-connected DNNs, TUTOR, on an average, reduces the need for data by 5.9× (geometric mean), improves accuracy by 3.4%, and reduces the number of parameters (floating-point operations) by 4.7× (4.3×) (geometric mean). Thus, TUTOR enables a less data-hungry, more accurate, and more compact DNN synthesis

QAIS-DSNN: Tumor Area Segmentation of MRI Image with Optimized Quantum Matched-Filter Technique and Deep Spiking Neural Network

Tumor segmentation in brain MRI images is a noted process that can make the tumor easier to diagnose and lead to effective radiotherapy planning. Providing and building intelligent medical systems can be considered as an aid for physicians. In many cases, the presented methods’ reliability is at a high level, and such systems are used directly. In recent decades, several methods of segmentation of various images, such as MRI, CT, and PET, have been proposed for brain tumors. Advanced brain tumor segmentation has been a challenging issue in the scientific community. The reason for this is the existence of various tumor dimensions with disproportionate boundaries in medical imaging. This research provides an optimized MRI segmentation method to diagnose tumors. It first offers a preprocessing approach to reduce noise with a new method called Quantum Matched-Filter Technique (QMFT). Then, the deep spiking neural network (DSNN) is implemented for segmentation using the conditional random field structure. However, a new algorithm called the Quantum Artificial Immune System (QAIS) is used in its SoftMax layer due to its slowness and nonsegmentation and the identification of suitable features for selection and extraction. The proposed approach, called QAIS-DSNN, has a high ability to segment and distinguish brain tumors from MRI images. The simulation results using the BraTS2018 dataset show that the accuracy of the proposed approach is 98.21%, average error-squared rate is 0.006, signal-to-noise ratio is 97.79 dB, and lesion structure criteria including the tumor nucleus are 80.15%. The improved tumor is 74.50%, and the entire tumor is 91.92%, which shows a functional advantage over similar previous methods. Also, the execution time of this method is 2.58 seconds

MHDeep: Mental Health Disorder Detection System based on Body-Area and Deep Neural Networks

Mental health problems impact quality of life of millions of people around the world. However, diagnosis of mental health disorders is a challenging problem that often relies on self-reporting by patients about their behavioral patterns and social interactions. Therefore, there is a need for new strategies for diagnosis and daily monitoring of mental health conditions. The recent introduction of body-area networks consisting of a plethora of accurate sensors embedded in smartwatches and smartphones and edge-compatible deep neural networks (DNNs) points towards a possible solution. Such wearable medical sensors (WMSs) enable continuous monitoring of physiological signals in a passive and non-invasive manner. However, disease diagnosis based on WMSs and DNNs, and their deployment on edge devices, such as smartphones, remains a challenging problem. To this end, we propose a framework called MHDeep that utilizes commercially available WMSs and efficient DNN models to diagnose three important mental health disorders: schizoaffective, major depressive, and bipolar. MHDeep uses eight different categories of data obtained from sensors integrated in a smartwatch and smartphone. These categories include various physiological signals and additional information on motion patterns and environmental variables related to the wearer. MHDeep eliminates the need for manual feature engineering by directly operating on the data streams obtained from participants. Since the amount of the data is limited, MHDeep uses a synthetic data generation module to augment real data with synthetic data drawn from the same probability distribution. We use the synthetic dataset to pre-train the weights of the DNN models, thus imposing a prior on the weights. We use a grow-and-prune DNN synthesis approach to learn both the architecture and weights during the training process. We use three different data partitions to evaluate the MHDeep models trained with data collected from 74 individuals. We conduct two types of evaluations: at the data instance level and at the patient level. MHDeep achieves an average test accuracy, across the three data partitions, of 90.4%, 87.3%, and 82.4%, respectively, for classifications between healthy and schizoaffective disorder instances, healthy and major depressive disorder instances, and healthy and bipolar disorder instances. At the patient level, MHDeep DNNs achieve an accuracy of 100%, 100%, and 90.0% for the three mental health disorders, respectively, based on inference that uses 40, 16, and 22 minutes of data from each patient

CovidDeep: SARS-CoV-2/COVID-19 Test Based on Wearable Medical Sensors and Efficient Neural Networks

The novel coronavirus (SARS-CoV-2) has led to a pandemic. The current testing regime based on Reverse Transcription-Polymerase Chain Reaction for SARS-CoV-2 has been unable to keep up with testing demands, and also suffers from a relatively low positive detection rate in the early stages of the resultant COVID-19 disease. Hence, there is a need for an alternative approach for repeated large-scale testing of SARS-CoV-2/COVID-19. The emergence of wearable medical sensors (WMSs) and deep neural networks (DNNs) points to a promising approach to address this challenge. WMSs enable continuous and user-transparent monitoring of physiological signals. However, disease detection based on WMSs/DNNs and their deployment on resource-constrained edge devices remain challenging problems. To address these problems, we propose a framework called CovidDeep that combines efficient DNNs with commercially available WMSs for pervasive testing of the virus and the resultant disease. CovidDeep does not depend on manual feature extraction. It directly operates on WMS data and some easy-to-answer questions in a questionnaire whose answers can be obtained through a smartphone application. We collected data from 87 individuals, spanning three cohorts including healthy, asymptomatic (to detect the virus), and symptomatic (to detect the disease) patients. We trained DNNs on various subsets of the features automatically extracted from six WMS and questionnaire categories to perform ablation studies to determine which subsets are most efficacious in terms of test accuracy for a three-way classification. The highest test accuracy obtained was 98.1%. The models were also shown to perform well on other performance measures, such as false positive rate, false negative rate, and F1 score. We augmented the real training dataset with a synthetic training dataset drawn from the same probability distribution to impose a prior on DNN weights and leveraged a grow-and-prune synthesis paradigm to learn both DNN architecture and weights. This boosted the accuracy of the various DNNs further and simultaneously reduced their size and floating-point operations. This makes the CovidDeep DNNs both accurate and efficient, in terms of memory requirements and computations. The resultant DNNs are embedded in a smartphone application, which has the added benefit of preserving patient privacy